Endoscope employing structured light providing physiological feature size measurement

ABSTRACT

Disclosed herein are systems, methods, and structures providing accurate and easy to use size measurement of physiological features identified from endoscopic examination. In sharp contrast to the prior art, systems, methods, and structures according to the present disclosure employ structured light that advantageously enables size and/or distance information about lesions and/or other physiological features in a gastrointestinal (GI) tract. Advantageously, systems, methods, and structures according to the present disclosure are applicable to both capsule endoscopes and insertion endoscopes.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuing application of U.S. patent applicationSer. No. 15/927,856 filed 21 Mar. 2018, which in turn is acontinuation-in-part application of U.S. patent application Ser. No.14/884,788 filed 16 Oct. 2015 now issued on 3 Apr. 2018 as U.S. Pat. No.9,936,151, both of which are incorporated by reference as if set forthat length herein.

TECHNICAL FIELD

This disclosure relates generally to endoscopic examination of bodylumens and more specifically to endoscopes and endoscopic examinationemploying structured light to facilitate the accurate dimensionalmeasurement of lesions or other features observed during suchexamination.

BACKGROUND

As is known, endoscopes—including capsule endoscopes—allow clinicians tofind and identify lesions and other physiological features in agastrointestinal (GI) tract. Such capsule endoscopes are capsuleshaped—having a tubular body with end structures giving them theircapsule shape—and may advantageously be swallowed or taken into astomach by traversing the throat and esophagus with a voluntary muscularaction, as food, drink, or other substances. From the stomach, thecapsule proceeds through the intestines and subsequently exits.Subsequent diagnosis oftentimes includes an estimation of the size ofthe lesion/feature since any health risk posed by the lesion/feature andany subsequent treatment regime(s) often depend on its size. Forexample, adenomas and sessile serrated polyps in a colon are typicallycategorized as advanced precancerous lesions if they measure more than 1cm in diameter.

Despite the recognized importance of physiological feature sizemeasurement, contemporary endoscopes—particularly capsuleendoscopes—lack an accurate and easy to use method of size measurementfor such physiological feature(s). Accordingly, methods, systems, andstructures that provide or otherwise facilitate the size measurement ofsuch physiological features identified from endoscopic examination wouldrepresent a welcome addition to the art.

SUMMARY

An advance in the art is made according to aspects of the presentdisclosure directed to methods, systems and structures providingaccurate and easy to use size measurement of physiological featuresidentified from endoscopic examination.

In sharp contrast to the prior art, systems, methods, and structuresaccording to the present disclosure employ structured light thatadvantageously enables size and/or distance information about lesionsand/or other physiological features in a gastrointestinal (GI) tract.

Advantageously, systems, methods and structures according to the presentdisclosure are applicable to both capsule endoscopes and insertionendoscopes.

Viewed from one aspect, the present disclosure is directed to endoscopesystems including: a housing; at least one camera; a structured lightsource; and an array of microlenses that produces the structured light,the array of microlenses positioned such that light emitted from thestructured light source is collimated by the microlenses into an arrayof beams propagating in multiple directions.

Viewed from another aspect, the present disclosure is directed tomethod(s) for imaging a body lumen comprising: introducing an imagingapparatus into the body lumen; emitting, from the imaging apparatus,non-structured light into the body lumen; detecting, by the imagingapparatus, non-structured light reflected from anatomical features inthe body lumen; generating, by the imaging apparatus, one or morenon-structured light images from the detected non-structured light;projecting structured light into the body lumen; detecting structuredlight reflected from the anatomical features in the body lumen; andgenerating one or more structured light images from the detectedstructured light.

BRIEF DESCRIPTION OF THE DRAWING

A more complete understanding of the present disclosure may be realizedby reference to the accompanying drawing in which:

FIG. 1 is a schematic illustratively showing the generation ofstructured light with a shadow mask according to aspects of the presentdisclosure;

FIG. 2 is a schematic illustratively showing the generation ofstructured light with a shadow mask and collimating lenses according toaspects of the present disclosure;

FIG. 3 is a schematic illustratively showing the generation ofstructured light with a shadow mask and collimating lenses according toaspects of the present disclosure in which the width of the apertures Dis equal to the pitch (duty cycle is 100%);

FIG. 4 is a schematic illustratively showing integration of a micro lensarray (MLA) formed on/in a substrate according to aspects of the presentdisclosure;

FIG. 5 is a schematic illustratively showing integration of a micro lensarray (MLA) formed on/in a substrate with integrated aperture maskaccording to aspects of the present disclosure;

FIG. 6 is a schematic illustratively showing integration of a micro lensarray (MLA) formed on/in a substrate and subsequently embossed/molded oretched to form the lenses according to aspects of the presentdisclosure;

FIG. 6(A), and FIG. 6(B) are schematic diagrams showing illustrativemicrolens array (MLA) patterns in which: FIG. 6(A) shows a close packed(i.e., hexagonal or “honeycomb”) arrangement and FIG. 6(B) shows arectangular arrangement, both according to aspects of the presentdisclosure;

FIG. 7 is a schematic illustratively showing an optical elementpositioned after the MLA in the optical path such that overall field ofview (FOV) is increased according to aspects of the present disclosure;

FIG. 8 is a schematic illustratively showing a capsule endoscopeincluding structured light elements according to aspects of the presentdisclosure;

FIG. 9(A) and FIG. 9(B) are a schematic diagrams showing: FIG. 9(A)structured light elements according to the present disclosure includedin a contemporary insertion endoscope having structured light elementsand additional white light elements according to aspects of the presentdisclosure; and FIG. 9(B) structured light elements according to thepresent disclosure included in an alternative contemporary insertionendoscope having a structured light channel and elements and additionalillumination channel elements according to aspects of the presentdisclosure;

FIG. 10(A) and FIG. 10(B) are schematic diagrams illustratively showinga capsule endoscope including structured light elements and exhibiting apanoramic imaging system according to aspects of the present disclosure;

FIG. 11 is a schematic perspective view showing an illustrative MLAwherein microlenses of the array are arranged in concentric rings abouta center LED according to aspects of the present disclosure;

FIG. 12 is a schematic diagram illustrating an image captured by acamera—configured according to the present disclosure—the image being ofa planar surface with SL projected onto it according to aspects of thepresent disclosure;

FIG. 13 is a plot illustrating a responsivity spectrum for anillustrative sensor and the emission spectrum of an illustrativestructured light source included in an endoscope structure according toaspects of the present disclosure;

FIG. 14(A) and FIG. 14(B) are plots showing a sensor luma signal for R,G, and B pixels as a function of position x on the sensor for the casesFIG. 14(A) when an object is close to the endoscope and the irradianceon the sensor is high and FIG. 14(B) when an object is farther and theirradiance on the sensor is lower—according to aspects of the presentdisclosure;

FIG. 15 is a schematic diagram of an illustrative capsule endoscopestructure employing structured light plot inside a body lumen and apolyp according to aspects of the present disclosure;

FIG. 16 is a schematic diagram of an illustrative endoscopeconfiguration having an optical element following the MLA in an opticalpath according to aspects of the present disclosure; and

FIG. 17 is a schematic diagram of an illustrative computer system thatmay execute methods according to aspects of the present disclosure.

Illustrative embodiments are described more fully by the Figures anddetailed Description. Embodiments according to this disclosure may,however, be embodied in various forms and are not limited to specific orillustrative embodiments described in the Drawing and detailedDescription.

DESCRIPTION

The following merely illustrates the principles of the disclosure. Itwill thus be appreciated that those skilled in the art will be able todevise various arrangements which, although not explicitly described orshown herein, embody the principles of the disclosure and are includedwithin its spirit and scope.

Furthermore, all examples and conditional language recited herein areprincipally intended expressly to be only for pedagogical purposes toaid the reader in understanding the principles of the disclosure and theconcepts contributed by the inventor(s) to furthering the art and are tobe construed as being without limitation to such specifically recitedexamples and conditions.

Moreover, all statements herein reciting principles, aspects, andembodiments of the disclosure, as well as specific examples thereof, areintended to encompass both structural and functional equivalentsthereof. Additionally, it is intended that such equivalents include bothcurrently known equivalents as well as equivalents developed in thefuture, i.e., any elements developed that perform the same function,regardless of structure.

Thus, for example, it will be appreciated by those skilled in the artthat any block diagrams herein represent conceptual views ofillustrative circuitry embodying the principles of the disclosure.Similarly, it will be appreciated that any flow charts, flow diagrams,state transition diagrams, pseudo code, and the like represent variousprocesses which may be substantially represented in computer readablemedium and so executed by a computer or processor, whether such computeror processor is explicitly shown.

The functions of the various elements shown in the Drawing, includingany functional blocks labeled as “processors”, may be provided usingdedicated hardware as well as hardware capable of executing software inassociation with appropriate software. When provided by a processor, thefunctions may be provided by a single dedicated processor, by a singleshared processor, or by a plurality of individual processors, some ofwhich may be shared. Moreover, explicit use of the term “processor” or“controller” should not be construed to refer exclusively to hardwarecapable of executing software, and may implicitly include, withoutlimitation, digital signal processor (DSP) hardware, network processor,application specific integrated circuit (ASIC), field programmable gatearray (FPGA), read-only memory (ROM) for storing software, random accessmemory (RAM), and non-volatile storage. Other hardware, conventionaland/or custom, may also be included.

Software modules, or simply modules which are implied to be software,may be represented herein as any combination of flowchart elements orother elements indicating performance of process steps and/or textualdescription. Such modules may be executed by hardware that is expresslyor implicitly shown.

Unless otherwise explicitly specified herein, the FIGs comprising thedrawing are not drawn to scale.

By way of some additional background, we again note that—despite theimportance of measuring the size(s) of physiological feature(s) that maybe identified from endoscopic examination—contemporaryendoscopes—including capsule endoscopes—do not adequately provide suchsize measuring capability. Note that for brevity, we may interchangeablyuse the terms “features” or “lesion” to describe such physiologicalfeatures. We note further that for the purposes of this disclosure andclaims, such feature or lesion is simply an object or point of interestin a field of view and no nomenclature used herein is to be consideredas limiting.

As will be known and readily understood by those skilled in the art,size measurement/estimation of an endoscopic camera image is error proneas the apparent size of an object or physiological feature to bemeasured depends upon its magnification, which in turn depends on itsdistance from the camera—which is generally not known. When an object isclose to a camera, i.e., small conjugate distance, as is necessarily thecase for in-vivo imaging, small changes in object distance produce largechanges in magnification. Moreover, for the wide-angle lenses employedin endoscopes, lens distortion produces magnification variation acrossthe camera field of view.

Those skilled in the art will readily appreciate that a tool (i.e.,scale, forceps, or other object of a known size) of some sort may beused during an endoscopic examination as a size reference by positioningit sufficiently proximate to a lesion and viewing the tool and thelesion to provide a size reference to better estimate/determine the sizeof the lesion. As will be further appreciated, such a procedure may betime consuming, difficult or impossible for certain lesion positions inthe bowel, not sufficiently accurate, present increased procedural riskof mucosal damage, and does not automatically record a measurement intoa storage medium as part of the procedure record. Moreover, such toolsare not available for capsule endoscopic procedures.

Of course, an endoscope including such tool that measures distance wouldenable a fast, simple, and objective measurement and recording of thesize of objects in the gastrointestinal (GI) tract observed duringendoscopic examination and would be a welcome addition to the art.

It is noted at this point that various electronic technologies have beendeveloped for measuring the distance of objects, including radar,ultrasound, sonar, echo-location, lidar, holography, stereo-vision,depth-from-shading, time-of-flight, optical coherence tomography,confocal microscopy, and structured light. Many of these technologiesrequire large, complicated, expensive, and power-hungry systems,methods, and structures. Optical time-of-flight measurements—includinglidar—are challenging to employ for short object distance(s) because thetime of flight is quite brief and therefore millimeter scale distanceresolution is difficult to achieve. Optical coherence tomography (OCT)and confocal microscopy have been used in endoscopes and proceduresemploying same but are insufficiently miniaturized to provide utilityfor non-tethered, capsule endoscope applications. Finally, many of thesenoted technologies require sensor hardware that operates separately fromoptical white-light (WL) cameras employed by gastroenterologists orothers (who employ endoscopes—i.e., endoscopists) to identify lesionsand other features, making the correspondence between camera-image dataand 3D-sensor data challenging.

Advantageously—and as will be readily appreciated by those skilled inthe art and according to aspects of the present disclosure—3D data maybe acquired by triangulation between an endoscope camera and a projectorof structured light (SL). As will be further appreciated, such anapproach leverages the camera—thereby reducing any extra hardwarerequired and simplifying the establishment of a correspondence betweenwhite light image data and depth measurements.

As is used herein, structured light is spatially patterned so that anobject space is illuminated with a pattern of known geometry in 3Dspace. Structured light involves a process of projecting a known pattern(oftentimes including grid or horizontal bar pattern elements, orrandom, pseudorandom, or semi-periodic elements) onto a scene (e.g., anyfield of view of an endoscope camera). The way(s) in which patternelements deform when striking surfaces allows the determination of thedepth and surface information of objects in the scene, as used bystructured light 3D scanning systems. Advantageously, structured lightmay employ invisible (or imperceptible) structured light withoutinterfering with other light-based vision systems and/or processes forwhich the projected pattern may be confusing. Illustrative invisiblestructured light includes the use of infrared light or of sufficientlyhigh frame rates alternating between different patterns, i.e., oppositepatterns.

By way of some specific examples, we note that structured light mayemploy an array of beams—emitted from one or more centers of projection(COP)—creating a grid of spots, lines, or other patterns on anilluminated surface. For triangulation, as will be known and understoodby those skilled in the art, the COPs of the structured lightprojections must not be co-located with the COP of a camera imaging thepattern/surface.

Structured light may be generated by an image protector (structuredlight projector) that projects an image from a spatial light modulatoror “slide” onto a surface. With respect to an in-vivo application, suchsurface may generally include biological materials including mucosaand/or lesions and/or other physiological features. However, opticalefficiency of such image projectors decreases with the size of thesystem. For a single aperture image projector, flux is proportional tothe focal-length squared. Since endoscopes typically exhibit a largefield of view (FOV), e.g., 160°, it is difficult for image projectors tocover such a large FOV. Similarly, alternative technologies includingminiature optical scanners using—for example—micro-electro-mechanicalsystems (MEMS) mirrors—cannot easily cover such a large FOV. Stillalternative technologies such as diffraction gratings or holograms thatoperate by passing coherent light through a diffractive optical element(DOE)—while they may generate spatial patterns—such patterns however,only exhibit sufficient contrast in the diffraction far-field—typicallyat a distance of at least 1 cm from the DOE—and uniform coverage of aFOV exceeding 60° is difficult to achieve from a single source. Stillanother technological approach for generating structured light employingfocusing lens(es) positioned at a focal length beyond the DOE producesimages in the diffraction far-field at a distance equal to twice thefocal length of the lens(es) from the DOE but results in a reducedcoverage area (FOV) of the image.

Given these and other difficulties, the present disclosure is directedto systems, methods, and structures for the generation of structuredlight in a constrained spatial volume, exhibiting sufficiently low powerconsumption and low cost yet well suited for the illumination of objectdistances in the range of millimeters and beyond. As we shall show anddescribe, such systems, methods, and structures according to the presentdisclosure are particularly attractive to endoscopic applications andmore particularly in-vivo capsule endoscopes. Notwithstanding suchattractiveness, systems, methods, and structures according to thepresent disclosure advantageously exhibit applicability to otherapplications as well.

As will become apparent to those skilled in the art and for the purposesof presenting an elementary analogy, systems, methods, and structuresaccording to the present disclosure employ a variation of a basicconcept of casting shadows with a shadow mask. More particularly—andaccording to the present disclosure—light is passed through an array ofapertures including micro-lenses that collimate the light into an arrayof beams—the intensity of which decreases less rapidly with distancethan light passing through apertures without collimating lenses. Sinceeach beam is independently collimated, the beam angles may vary widelyto cover a larger FOV. Additionally, mirrors, lens(es), or other opticalelements positioned beyond the micro-lenses may be employed to redirectsome—or all—of the beams and increase FOV—as we shall show and describein greater detail later in this disclosure.

At this point we note that those skilled in the art will readilyunderstand and appreciate that a collimator is a device that narrows oneor more beam(s) of light. As used herein, collimate means to narrow orcause direction(s) of light to become more aligned in a specificdirection (i.e. that light rays describing the beam become moreparallel), however, it does not mean that no divergence or convergenceoccurs in the collimated beam. It is possible that collimation mayresult in beams that have a smaller spatial cross section.

As will become further apparent to those skilled in the art, much of thedisclosure presented herein is illustratively presented in the contextof capsule endoscopes. The disclosure is not so limited. Systems,methods, and structures according to the present disclosure contemplatecapsule and insertion-tube type endoscopes, as well as otherinstrumentation that may benefit from size and/or distance determinationof objects of interest in a scene.

Turning now to FIG. 1, there is shown a schematic diagram of anillustrative apparatus generating structured light using a shadow maskaccording to aspects of the present disclosure. As may be observed fromthat figure, light is emitted from a source through an aperture (sourceaperture) having a diameter a. A shadow mask having an array ofapertures (mask apertures) of width D is positioned a distance L fromthe source aperture through which the source light was emitted.Accordingly, and as shown illustratively in this figure, a divergenceangle of light in a beam beyond the mask aperture(s)—ignoringdiffraction on the axis with the source—is defined by:θ=tan⁻¹((a+D)/2L).

As will be further appreciated by those skilled in the art, as thedivergence angle increases, the greater the intensity of the lightdecreases with distance, thus requiring greater dynamic range in animage to adequately detect the presence and location of any projectedpatterns (spots, etc.) on both distant and near objects. As used hereinand as generally understood by those skilled in the art, dynamic rangedescribes a ratio between maximum and minimum measurable lightintensities (e.g., black and white).

Reducing a and D (narrowing the mask apertures) reduces the divergenceat the expense of throughput. Also, diffraction limits how much thedivergence can be reduced by reducing D. Also, for projected spots to bedistinguishable from neighboring spots, an aperture duty cycle of atleast approximately 50% is required (i.e., the mask aperture pitch is atlast 2D). Note that the shadow mask shown in the figure exhibitingsquare mask apertures must be substantially 50% opaque along axes inboth lateral directions such that only approximately 25% of incidentlight striking the mask is passed. Such criteria are less for circularmask apertures.

With reference now to FIG. 2, there is shown a schematic diagram of anillustrative apparatus generating structured light using a shadow maskand a collimating lens positioned in each mask aperture according tofurther aspects of the present disclosure. If such collimating lens(es)are positioned in the mask aperture as shown, beam divergence—ignoringdiffraction—is reduced to:θ=tan⁻¹(a/2L).

While not yet specifically shown in the figures, a light source mayadvantageously include a “point-source” light-emitting diode (LED),which is an LED having a small aperture.

In an illustrative embodiment, a point-source LED exhibits a structuresimilar to that of a standard LED. However, the light emitted therefromis emitted through a well-defined, (often circular) area, typically 25μm˜200 μm in diameter. The light so produced will appear as a “spot”producing a narrow viewing angle. As will be appreciated, such apoint-source LED may eliminate or change the requirements of the sourceaperture (and any source mask having the source aperture) illustrativelyshown. (In such case, a is equivalent to an aperture diameter of thepoint source.) Typically, a lateral current-confinement structure isincluded in an LED such that an area in which electrons and holesrecombine therein is not much larger than the aperture. The aperture mayan opening in a metal layer on the surface of the LED.

Of course, a source employed in systems, structures, and methodsaccording to the present disclosure may also comprise a laser, includinga vertical-cavity surface-emitting laser (VCSEL) which may have anaperture of 10 μm or less, and is known to be much more efficient that apoint-source LED. Unfortunately, if such a laser is highly coherent, thegenerated structured light may include spurious interference patternsand speckle noise.

For a point-source LED, a would typically be in the range of 0.050 mm to0.20 mm (e.g., 0.080 mm or 0.10 mm) and L would typically be in therange of 1 mm to 10 mm. As such, for a=0.80 mm and L=4.0 mm, 0=6°. Solong as D>a, the beam divergence θ is less than the beam separationangle ϕ and the duty cycle of any spots projected on an object decreaseswith object distance, even if the lens duty cycle is 100% (i.e., thepitch equals D). Such a configuration is shown schematically in FIG. 3.As will be appreciated by those skilled in the art, a large apertureduty cycle increases the throughput relative to a shadow mask having nocollimating lenses.

At this point, note that a lens array such as that shown in the figuresmay be a micro lens array (MLA) formed or otherwise positioned on atransparent substrate. Such configuration is illustratively shown in theschematic diagram of FIG. 4.

With continued reference to that figure, it is noted that the substratemay be constructed from any of a variety of known materials includingglass, silica, polymeric, or other transparent material(s). Likewise,lens(es) may be constructed from any suitable material and formed byembossing, molding, or lithography with photoresist reflow and/oretching or any other known technique. The lens array may reside on glassframework, which in turn may be affixed or otherwise integrated with theoverall substrate. The lenses may be integrated onto the surface of thesubstrate facing the source or on the opposite side or integrated intothe body of the substrate. Note that if the lenses are positioned on anopposite side of the substrate with respect to a light source, the focallengths are larger for a same substrate position and thickness resultingin a reduced beam divergence.

Note further that each individual lens in the array has an opticalaxis—an axis of symmetry for the lens. For each lens, a chief ray passesfrom the source and intersects the optical axis at the lens entrancepupil. The chief ray and the optical axis lie in a tangential plane, andthe chief ray also lies in a sagittal plane perpendicular to thetangential plane.

With reference to FIG. 5, it may be observed that a surface of the lensmaterial residing outside of lens clear apertures (CAs) may not exhibita shape or quality (shape or otherwise) effective to collimate lightwith low aberrations. Advantageously, a mask may be employed to blockany light that would otherwise traverse the MLA substrate outside of theCAs and reduce the contrast of the structured light projection—as shownschematically in FIG. 5. The mask may be constructed from a sheet ofmaterial or a coated substrate positioned immediately before or afterthe microlens array in the optical path depicted in the figure.Additionally, the mask may be an opaque coating applied to the surfaceof the MLA. For example, a black chrome or other opaque material may bedeposited or otherwise applied on the surface of the lens array andpatterned—using photo etching or other known methods—into aperturesappropriately aligned with the individual lenses. Alternatively, blackchrome or other opaque material may be applied and patterned on a glasssubstrate. Subsequently, a thin polymer layer may be applied to thesubstrate on the chrome (or other suitable material) and a mold appliedresulting in an embossed lens pattern such that the lenses are alignedto the black chrome apertures. Illustrative structures resulting fromsuch formation are shown schematically in FIG. 6.

We note that microlenses may be configured in alternative arrangements(patterns) according to aspects of the present disclosure. For example,FIG. 6(A), and FIG. 6(B) are schematic diagrams showing illustrativemicrolens array (MLA) patterns in which: FIG. 6(A) shows a close packed(i.e., hexagonal or “honeycomb”) arrangement and FIG. 6(B) shows arectangular arrangement, both according to aspects of the presentdisclosure. As may be observed from those figures, these illustrativearrangements result in substantially no space between adjacentlens'—consequently a mask layer may be unnecessary for thesearrangements. We note that in practice, however, the transition from onelens sag to the next adjacent one will not be perfectly sharp andtherefore some light traversing through these transition regions willnot be well collimated by the lens array. Notwithstanding such lack ofcollimation, a fraction of light so aberrated may be tolerably low sothat no correction may be necessary.

With these MLA configurations in mind, we note the field-of-viewhalf-angle covered by the structured light source is approximately

${\varphi \approx {\sin^{- 1}\left( \frac{w}{f} \right)}};$where w is the width of the MLA and f is the tangential focal length ofthe lens at the edge of the array. To minimize beam divergence, f, andhence the distance L from the source to the MLA, φ should be as large asavailable space permits.

To increase FOV, w must be increased relative to f. However, the cost ofMLA scales with its area and hence w². Also, as the angle of incidencefor light through the MLA increases, lens aberrations, Fresnel losses,pupil distortion, and reduced light-emitting diode (LED) intensity(since LED intensity drops with angle) all become increasinglyproblematic.

Advantageously, the FOV may be increased without increasing w/f byplacing an optical element after the MLA which increases the divergenceof transmitted beams. FIG. 7 is a schematic illustratively showing anoptical element positioned after the MLA in the optical path such thatoverall field of view (FOV) is increased according to aspects of thepresent disclosure.

With reference now to that figure, there it shows a negative powerrefractive lens L1 positioned such that it follows the MLA in theoptical path. In an illustrative configuration, a source is positionedon the optical axis of L1 and the lens array is perpendicular to thisaxis, but such arrangement advantageously need not be so exact.

As illustrated in that figure, rays are shown for two microlenses (MLs)of the MLA. Since L1 diverges the beams, the ML positive power iseffectively increased such that they focus light from the source withfinite conjugate to a point beyond L1. Additionally, the MLs exhibitdifferent curvatures in the tangential and sagittal planes such that thebeams in object space beyond L1 are approximately equally wellcollimated in both directions. Each ML collimates the light, making therays that pass through it more parallel but somewhat convergent, and L1further collimates the light, reducing or eliminating the convergence.Also, the ML curvatures vary with lens center position relative to theoptical axis. To make beam widths substantially equal, the CA of MLs mayincrease with distance from the optical axis. Additionally, a differenttype of optical element may be used in place of or in addition to L1 toincrease the structured light (SL) FOV such as a Fresnel lens, adiffractive element, one or more mirrors, or one or more prisms.Finally, note that the FOV 2ϕ covered by the SL could be over 180°, orless than 180°, including—for example—160°, 140°, 120°, 100°, 80°, and60°.

Turning now to FIG. 8, there is shown in schematic form an illustrativecapsule endoscope including structured light elements according toaspects of the present disclosure. As depicted in that figure, astructured light source—which may include a light emitting diode orother suitable emitter/source including a laser—is positioned on aprinted circuit board (PCB) and located within a body of the capsule.Shown in the figure, the structured light elements are positionedadjacent to a camera which—as will be readily appreciated by thoseskilled in the art—may generally include one or more sensors and/orimaging structures along with optical elements as well as any electronicand mechanical components. Note that many configuration possibilitiesare possible and contemplated by systems, methods, and structuresaccording to the present disclosure in addition to those specificallyshown in the figures. As an illustration, the camera and/or sensor maybe mounted on the same PCB—or a different one—depending upon theparticular configuration employed. Additional/other/alternative lightsources may likewise be positioned around the camera.

For example, additional light sources may be placed around the camera ina ring or other arrangement. Additional structured light elements maylikewise be positioned around the camera, and interposed between thelight sources or, in any other arrangement that produces a desiredillumination and structured light emissions and/or patterns.

As illustratively shown in FIG. 8, the capsule endoscope includes thestructured light element having a structured light source (LED, or otherlight emitter—preferably point source), a microlens array and anadditional optical element (lens) L1 shown illustratively exhibiting anegative optical power. Such a lens is advantageously positioned afterthe MLA thereby increasing the FOV coverage of the structured lightelement (SL projector/generator). Advantageously, L1 may be part of amolded (or other fabrication technique including 3D printing) structurethat includes other functional optical elements and may extend at leastpart-way around the perimeter of the camera. Contemplated otherfunctional optical elements that are not specifically shown in thisillustrative schematic figure include other lenses, other/alternative SLsources and/or diffusers for additional/alternative lightsources—including white light. Shown further in this illustrative figurethe camera has a center of projection COP1 and the SL projector has acenter of projection COP2—which is offset from COP1.

With continued reference to FIG. 8, it may be observed that theillustrative capsule endoscope includes a battery or other power source,a controller, memory, and a transmitter. The capsule itself includes acylindrical body, with a dome-shaped window at one end and a dome-shapedbody element at an opposite end. Note that while we have used the word“window” to describe the dome-shaped element forming/closing one end ofthe capsule body, such dome-shaped window may be made from any suitablematerial that is compatible with the cylindrical body and issufficiently transparent to light employed. Note further that while thedome-shaped window is only shown at one end, contemplated configurationsmay include dome-shaped windows at both ends and duplicate/supplementalimaging systems/structured light elements/optical elements may bepositioned therein.

At this point we note that a capsule endoscope such as that according tothe present disclosure is swallowable (ingestible) by a human and assuch will exhibit a size of approximately 4 cm or less in length andapproximately 2 cm or less in diameter. Such a capsule may beconstructed from any of a number biocompatible materials that survive atrip through a digestive tract without comprising components containedwithin. Additionally, and as will be readily appreciated by thoseskilled in the art—at least portions of such capsules—in addition toexhibiting suitable biocompatibility—will also exhibit suitable opticalproperties. Once swallowed (ingested), capsules will pass through thedigestive tract via physiological processes, including peristalsis.

As those skilled in the art will readily appreciate, additionalhardware/electronic/optical components and additional software executedby the controller or other comparable structure(s) are contemplated.Operationally, image data stored in memory and then transmitted by thetransmitter to external storage and/or processing systems. In certainillustrative embodiments, the memory may include longer-term, orarchival storage in which image data is stored until the capsule islater retrieved after being excreted or otherwise removed from apatient. In yet other illustrative embodiments, the transmittertransmits data wirelessly through the patient body to an ex vivoreceiver, for example, by radio, ultrasound, human-body electricalconduction, or optically. Advantageously, a general apparatus like thatillustrated in FIG. 8 may be positioned on/within the tip of atraditional endoscope insertion tube rather than in a capsule endoscopeas illustratively shown in FIG. 8. Such apparatus—as we shall show—mayinclude combinations of those elements comprising the structureillustrated in the figure.

FIG. 9 is a schematic diagram showing an illustrative configuration of acontemporary insertion endoscope and insertion tube including structuredlight elements and additional white light elements and camera accordingto aspects of the present disclosure. As will be readily appreciated bythose skilled in the art, the insertion tube will be passed through abody orifice or incision made therein. Once positioned inside the body,the camera system may capture images of an interior body lumen. Notethat the illustrative structure shown in this figure is substantiallythe same as that shown earlier in FIG. 8. Of course, depending upon theparticular endoscope design, various configurations of such an endoscopeincluding structured light according to the present disclosure are wellwithin the scope of this disclosure even where not specifically shown inthis figure.

With simultaneous reference now to FIG. 8 and FIG. 9(A) and FIG. 9(B),we note that these figures illustrate three SL beam axes passing fromthe source, through the MLA, through lens L1 which deflects off-axisbeams into larger angles thereby increasing the FOV, through the capsulehousing and onto the mucosa (lumen wall) producing illuminated spots orpatterns substantially centered at points A, B, and C in a cavity(lumen) of a human or other body. Light scattered from spots at pointsA, B, and C are collected and imaged in/by the camera.

Note that with reference to these three figures, systems, methods, andstructures according to the present disclosure are shown in threedifferent configurations—while sharing many aspects of this disclosure.FIG. 8 shows an illustrative capsule endoscope according to aspects ofthe present disclosure. FIG. 9(A) shows an illustrativeinsertion-tube-type endoscope according to aspects of the presentdisclosure, and FIG. 9(B) shows an illustrative insertion-type endoscopeaccording to aspects of the present disclosure having a flat, non-domedend. Note further that with respect to the illustrative configurationshown in FIG. 9(B), the white, non-structured light is not generated atthe distal end, rather it is generated elsewhere and conveyed to the endvia an illumination channel. We note that such illuminationchannel/remote source may be replaced/supplemented by the LED or othersource(s) disclosed. Likewise, the SL source is not shown in the distalend of the insertion tube. Rather the SL source is elsewhere, forexample at or beyond the proximal end, and the SL is conveyed throughthe SL channel, such as an optical fiber or other lightguide, andemitted through an aperture (e.g. the end of the lightguide) and thenpasses through the MLA and L1. The SL source could instead be placed atthe location of the SL aperture similarly to the example of FIG. 9(A),eliminating the need for the SL channel.

The white light source may be activated during an image sensorintegration period of a same frame such that both SL and white lightilluminate a captured image. Advantageously, the SL source may exhibitan optical spectrum that is different than that exhibited by the whitelight source such that it has a distinguishable color in images thatinclude white light illumination. For example, the spectrum may benarrower such as that of a red, green, amber, or blue LED. The spectrumcould—for example—fall outside the white light spectrum such as in theinfrared (IR) region of the electromagnetic spectrum. Of course, animage sensor may include pixels with color filters that have a highertransmittance for the SL than for the white light. For example, pixelsthat are transmissive to IR and which absorb or otherwise block whitelight may be included on a sensor to advantageously detect IR structuredlight.

FIG. 10(A) and FIG. 10(B) show schematic diagrams of illustrativecapsule section(s) (or specifically configured insertion-typeendoscopes) including a panoramic imaging system and structured lightelements. In these illustrative embodiments depicted, the panoramicimaging system may include—for example—four camera systems each facingthe cylindrical—or tubular—wall of the capsule (or insertion endoscope).As will be readily appreciated by those skilled in the art—the actualnumber of camera systems employed in a particular configuration may bedifferent from that shown in the figures namely, two, three, four, ormore. Even a single camera system, typically including a panoramicimaging capability, would be useful in particular configurations.

Note that while FIG. 10(A) and FIG. 10(B) show cross sections of oneside of an illustrative capsule, such capsules may advantageouslyexhibit a mirror-symmetry about a longitudinal axis so only one of four(in these examples) camera systems is illustrated. A point-source LED isshown positioned on or near the longitudinal axis of the capsule,displaced in the longitudinal direction from the camera.

Operationally, and as noted previously, light from the LED passesthrough a microlens array (MLA). In one illustrative embodiment, lensescomprising the MLA are centered on rings concentric with the LED, asshown illustratively in FIG. 11. As depicted in that figure, the lensarray includes an opaque mask layer with apertures that block lightincident on the array from the LED which is outside the Lens CAs.

Referring again to FIGS. 10(A) and 10(B), it may be observed (in FIG.10(A)) the beam path for one lens of the MLA on one ring while in FIG.10(B) it may be observed another beam path for another lens on anotherring of the same MLA. In FIG. 10(A), the beams from one of the rings aredeflected by mirror M1—which may advantageously be an annular mirrorreflecting a set of beams, all of which pass through the same MLA ring.Note that M1 may advantageously exhibit a conical, spherical, asphericalor other shape as necessary.

As will be appreciated, mirror M1 directs (reflects) light beamsout—through the tubular wall of the capsule housing. Relative to theaxis of the light source—perpendicular to the MLA—M1 increases theangular field of view of the structured light beyond 180°. For example,the FOV may be 200°, 220°, or 240°. The mirror M1 reflection effectively“creates” a virtual source on the optical axis of the source that isshifted closer to the camera than the source. In FIG. 10(A), the opticalaxis of the source is shown as the longitudinal axis of the capsule.

To extract depth information from an image of the structured lightcaptured by the camera system, the camera center of projection (COP) andthe virtual source must be physically separated. As depicted in FIG.10(A), the separation is more in the transverse direction than thelongitudinal direction. Note that if multiple beams are deflected by M1and M1 is symmetrical about the source, the virtual source correspondsto a COP of the SL projector.

Turning now to FIG. 11 there is shown a schematic diagram of anillustrative microlens array (MLA) in which the individual lenses of thearray are arranged in substantially concentric circles. Morespecifically, the individual lenses are arranged on one of five,substantially concentric, circular rings. As configured in thisillustrative arrangement, the rings are further arranged in four (4)distinct sections, azimuthally aligned to four side-facing cameras in acapsule (or alternatively, another endoscope or imaging device/structureincluding a single or multiple cameras with single or multiple imagerswith at least one camera/imager associated with each individualsection).

Note that the lens CAs are defined by a patterned black opaque layer ofmaterial such as black chrome. The clear(er) apertures are shown asbeing elliptical—although they could be other shapes including circularor rectangular—among others. The long axis of the oblong lenses lies inapproximately tangential planes. The projection of the aperture onto aplane perpendicular to the chief ray is foreshortened. The oblongaperture compensates for the foreshortening to produce a moresymmetrical beam. Larger apertures pass more light than smallerapertures so that the relative intensity of the beams is controlled bysetting the aperture sizes.

At this point we note that the optical systems depicted illustrativelyin FIGS. 10(A), 10(B) and FIG. 11 are symmetrical—or approximatelysymmetrical—about the center of the system (e.g., capsule). However, asthose skilled in the art will readily appreciate that such “centersymmetry” is not necessary. For example, one or more sources may beconfigured such that they are centered off the longitudinal axis.Additionally, and/or alternatively, mirrors M1, M2, and M3 may notexhibit a rotational symmetry about the longitudinal axis and the lensesin the MLA may not lie on circular rings. For a capsule system havingfour cameras, it may be advantageous to implement four separate SLsystems—one for each camera. The four systems may advantageously employa common MLA substrate.

Returning our discussion of FIG. 10(B), it may be observed from thatfigure that a lens in the MLA reduces the divergence of light emitted bythe source and a mask layer (not specifically shown) filters lightoutside of the lens CAs. The one beam shown is representative of allbeams passing through one ring of the MLA. The beam is deflected bymirror M2 and then again by mirror M3. As will be readily appreciated,mirror M2 and mirror M3 may be annular mirrors.

As illustratively configured, the radial position of M2 inside thecapsule is less than that of M1 and the mirror apertures do not overlapsuch that both M1 and M2 may exist in the same system. After reflectionfrom M3, the beam passes out through the tubular wall of the capsule andilluminates mucosa within the field of view of the camera. Thecombination of M2 and M3 reflections results in a beam angle exiting thehousing similarly to the angle upon exiting the MLA.

As may be observed, however, the beam is displaced and appears toemanate from a virtual source (center of projection) further from thecamera than the source on the longitudinal axis. Light emitted directlyfrom the MLA would have been blocked by the camera and thereforeprevented from exiting the capsule. By moving the virtual source furtherfrom the camera than the source, the beam passes the camera withoutbeing blocked. As will be readily appreciated by those skilled in theart, this same approach may be employed to route beams around otherobstacles. Note that since during normal operation mucosa will contactthe capsule housing—it is desirable to position mirrors to direct thelight beams to cover as much of the housing within the FOV as possible.

FIG. 12 is a schematic diagram illustrating an image captured by acamera—configured according to the present disclosure—the image being ofa planar surface with SL projected onto it. The spots are produced bythe intersection of SL beams with an object plane. Each SL beam isillustratively produced by a microlens of the MLA. As the object movesfurther from the camera, the spot centroids move on converging epipolarlines, four of which are illustrated in the figure.

For example, one spot is at point A if the object is contacting theendoscope and at point B if the object is at the edge of the system'suseful range. Each spot moves on its own epipolar line (or curve, if thecamera image is distorted). To extract depth information about points inthe scene, the system identifies spots in the image and determines acorrespondence between the spots and the epipolar lines, which are basedon a camera model. The correspondence may be confounded if epipolarlines cross and a spot is detected near an intersection. Fortunately,standard, known techniques exist for resolving this and otherambiguities to establish the correspondence for all or most of thespots.

In particular, the position of each spot on its epipolar line isdetermined and this position establishes the depth of the object at thespot location in the image. The greater the number of spots, the betterthe resolution of the depth map that is determined. Since the size andbrightness of the spots also decrease with object distance—thesequantities may also be used to determine the distance of the object(s)onto which the spots are projected. Rather than identifying acorrespondence between individual spots and epipolar lines, the shape ofa surface with structured light projected thereon may be estimated byother known methods such as determining the SL pattern deformation bydetermining correlations between portions, comprising multiple spots, ofthe projected and imaged pattern with portions of the known undeformedpattern to determine a map of the pattern deformation from projection onthe surface.

While not yet specifically shown in the figures, it is neverthelessnoted that an endoscope system according to the present disclosure willgenerally include one or more computers (or equivalentsystems/structures/functionality) to receive image data from theendoscope system, process the data, display image data to a human—or“display” to an expert system—receive inputs from the human/expertsystem via an interface (e.g., GUI), present analysis results such asestimated object size, create or update a database of procedure data,and generate reports of the medical examination results.

The images—which include SL—are analyzed to extract information aboutthe distance of objects visualized in the images. This analysis mayadvantageously be performed in a batch mode for many or all SL imagesprior to presentation to a human reader or, to reduce processing time,it may be performed on select images that are flagged or otherwiseidentified by the reader or machine (e.g. expert system/algorithm(s))that operate on a set of images to determine images of interest. Forexample, either a reader or machine (algorithm) might identify apossible lesion in a particular frame, and then the depth informationfor that frame and/or neighboring frames is extracted from thestructured light image data.

We note that endoscope images are typically presented to a reader as aseries of still images such as a video. The reader views the videolooking for pathologies or other objects of interest. Frames containingsuch objects (frames of interest) may be selected and placed into a listor database of selected frames for the particular medical procedure.

As will be appreciated, some frames may include objects or regionswithin the overall image for which the reader desires a sizemeasurement. Such measurement may then be operationally indicated by thereader by any of a number of well-known computational tools includingGUIs. For example, the reader may select points on a periphery of aregion, draw a curve around periphery or draw a line across the region.

The system will then estimate the distance across the indicated region,for example between two designated points. If the image includesstructured light, it may be used to estimate the distance in objectspace of any objects/features of interest in the image. From thestructured light, a 3D model of a scene or portion of ascene—represented by the image—may be constructed. Such model may becoarse if the density of SL points is significantly less than the pixeldensity of the image.

While direct depth information may be available for those pixels thatlie near the centroids of the SL spots, it will not be for pixels thatlie between spots. As may be understood and readily appreciated by thoseskilled in the art, additional information in the image such as detectededges or depth-from-shading may be used to better estimate the depthinformation across the image. Advantageously, the depth may also beestimated in regions between the SL spots by interpolation from thecalculated depth at the SL centroids. Once the 3D coordinates for two ormore points demarcating an object are estimated, the cartesian distancebetween them in object space is determined.

A size measurement is typically displayed to a reader and recorded in adatabase. The functions of identifying and demarcating a region ofinterest may be performed by a machine-executed algorithm instead of ahuman reader, or the reader may work in conjunction with suchmachine-executed algorithm(s) to identify and demarcate such regions.

As will be readily appreciated by those skilled in the art, ifstructured light (SL) and white light (WL) illumination exist in thesame frame, a system must identify the structured light within a regularWL image background. Note that scattered SL may also produce backgroundlight. Note further that the structured light spots are known to line onepipolar lines. The location of these lines is determined from a cameramodel that may be based—at least partially—on camera and projectorcalibration. More particularly, the system looks for image features thatbest match the structured light in an expected shape, size, intensity,and color. Color—in particular—provides a useful way of distinguishingSL from WL when such SL color is sufficiently different from the WLillumination color.

We note that when reviewing a video or other set of images captured froman endoscopic system according the present disclosure, visiblestructured light in the video (or images) may be a distraction to thereviewer. Accordingly, various methods may be utilized to remove it froman image once such SL spots are identified.

More particularly, an estimate of an SL pixel signal may be subtractedfrom the image. Such estimate may be based on a model of the SLincluding its color. Accordingly, if a particular pixel is saturated ina color plane due to the SL or if the SL signal to be subtracted islarge, then the white light image signal in that color plane may beestimated based on the signal in other color planes.

For example, if the SL is predominately red, the red color plane may bereconstructed from the green and blue color plane data for pixels withinthe SL spot based on a statistical correlation between red, green, andblue color planes in the region of an image around that spot. Methodssuch as “in painting” may also be used to fill in missing image andcreate a continuous image appearance. To eliminate chromaerror—resulting from imperfect SL subtraction from the image—it may beadvantageously displayed as a gray-scale image. If the structured lightis in the IR and the structured light is detected by IR pixels, then anRGB image with minimal impairment by structured light is available.

Note that methods employed to subtract the SL from images are likely toleave some residual impact on the image quality. Therefore, it isdesirable for the SL to be captured in separate frames from the whitelight frames. As will be understood, the time difference(s) (separation)between the white light and SL frame(s) should be short enough such thatany change in scene is sufficiently small that depth informationdetermined in the SL frame(s) may be applied to the scene(s) in the WLframe(s) with minimal error.

To reduce the impact of any scene, change(s), the reviewer/reader maydemarcate an object in two or more WL frames. Then, the position andsize—in pixels—of the object in a SL frame temporally positioned betweentwo WL frames (i.e., interstitial) may be estimated by interpolationbetween the WL frames. If the object of interest appears in multipleframes, then the reviewer/reader (or machine system/algorithm) mayselect one or more frames in which to demarcate the object and aproximal SL frame in which to estimate the object size—based on anestimate of a rate of object movement—selecting frames with minimal—oracceptable—movement. We note that the amount of movement of objects in avideo or series of images may be estimated by known methods such as thecalculation of motion vectors. The motion metric on which frames may beselected may be based more on the motion of the particular object regionto be measured in the video than the overall motion of the entire scene.

Advantageously, a reviewer/reader or image recognition system algorithm(including any employing machine learning methodologies) may identify anobject in one frame that is of interest. Then, the reviewer—orsystem—may search for the same object in neighboring frames usingpattern recognition methods and/or algorithms. Then, from the set offrames including the object, one or more frames may be selected forobject-of-interest demarcation.

The frames may be selected based on multiple criteria such as a rate ofobject movement, the fraction of the object within an image boundary,and the quality of an image including factors such as exposure, motionblur, obscuration by fecal—or other—matter, and the presence of bubblesor turbidity. The algorithm may select particular frames and thereviewer/reader may confirm their suitability by making an entry usingthe GUI or other mechanism. In illustrative embodiments, selected framesmay have check boxes that are selected to keep or deselect frames. Thedemarcation of the object in these frames may advantageously beperformed manually using—for example—the GUI—or other mechanism—by thereviewer/reader or automatically by a system with confirmation orfine-tuning by the reviewer/reader. The size measurement based on thedemarcation and analysis of SL in the same or proximal frames may bepresented to a reviewer/reader on a screen. The measurement presentationmay include—for example—error bars, confidence intervals, or otherindicators of the accuracy of the measurement.

As will be readily appreciated by those skilled in the art, a video—orseries of images—captured by a capsule endoscope moving autonomouslythrough a GI tract has many image frames exhibiting redundantinformation since at times the capsule is not moving, moves retrograde,or dithers. The endoscope system may not display some frames that aredetermined to be redundant, i.e., showing the same features that aredisplayed in other frames. Also, multiple frames that captureoverlapping images of the scene may be stitched into a composite image.As will be understood, this reduction in frame number reduces the timeneeded to review the video.

When an object of interest—such as a lesion—is identified in one of thedisplayed frames the system may display a version of the video with allframes displayed—including those previously not displayed or thosecombined with other frames into stitched frames. The process of findingoptimal frames in which to demarcate the object and measure its size, asdescribed previously, can be applied to this larger set of frames. Thebest frame(s) for demarcating the object—based on criteria describedabove, or others—may be one of the frames that was not originallydisplayed.

Note that a region of interest for size measurement may not be fullyvisualized in a frame, especially if the region is large. However, twoor more frames containing portions of the region may be stitchedtogether so that all or most of the region is captured in the stitchedframe. The region may be demarcated in the stitched frame and cartesiandistance between demarcation points may be estimated based on thestructured light data in the frames stitched and/or interstitial frames.

As will be appreciated by those skilled in the art, capsule endoscopespresent some particularly unique imaging conditions. Accordingly, themagnification of objects imaged by an endoscope camera (whether acapsule or insertable) is larger if the object is immersed in a fluidrather than in air or other gas. Thus, the correct estimation of objectdepth using structured light depends on a knowledge of the immersingmedium.

During a colonoscopy, the colon is insufflated with gas. For capsuleendoscopy, the colon and other organs are preferably filled with clear(colorless) water. However, gas bubbles, including large pockets of gas,do exist in the lumen during capsule endoscopy. In a video or set ofimages, these bubbles may be recognized due to the appearance of brightspecular reflections of the illuminating light from the wet mucosalsurface and a change in mucosal color, relative to water immersedmucosa. Moreover, a meniscus is visible where the bubble boundarycrosses the capsule housing.

When a reviewer/reader or a machine algorithm has identified an objectfor size measurement, the reviewer may be queried to determine whetherthe object is immersed in a liquid or a gas. Since the object may bepartially in a liquid and partially in a gas, the reviewer/reader mayindicate a gas/liquid ratio for the immersion or may use a cursor tool(or other GUI or other mechanism) to mark areas that are in gas or inliquid. Of course, a computer implemented method/algorithm may performthese same functions.

The geometric model of the SL is modified based on the medium selected.Alternatively, a measurement based on a fixed single-medium model may bescaled ad hoc based on the selected medium. For example, if the SL modelassumes water immersion, but a fraction P of the diameter of a measuredobject is in gas (e.g., P=0.40), the size estimate may be adjusted by PMwhere M is the relative magnification in gas versus liquid. Finally, Mmay be a function of field position and estimated object distance andmay be based on an a-priori camera model and calibration.

At this point we note that endoscope calibration may advantageously beperformed during manufacturing. More particularly, an endoscope may bepresented with targets at known positions and orientations relative tothe endoscope camera. Some targets may include a pattern such as acheckerboard. The location of the features in the pattern in therecorded image can help determine a model of the camera including focallength, COP, pose, and distortion. Other calibration images are formedby illuminating SL from the endoscope onto one or more targets. Thesecalibration images help determine a model of the SL projection includingCOPs, pose, epipolar lines, and color.

Note that for a capsule endoscope, it is convenient to store calibrationdata as well as any images and/or parameters derived from images, in acapsule endoscope memory. This data can then be downloaded with any invivo data to a workstation for processing the in vivo data andextracting depth information from in vivo images using camera and SLmodels derived from—at least partially—the calibration data.Alternatively, the calibration data for a capsule can be associated witha capsule identifier, such as a serial number, and be stored in adatabase. Upon recovering the in vivo data and the identifier from thecapsule, the calibration data associated with the identifier can beretrieved from the database and use for processing the in vivo data.

Image sensors used in endoscopes oftentimes include a mosaic of colorfilters on the pixels. For example, a sensor may have red (R), green(G), and blue (B) pixels with responsivity spectra as illustrativelyshown in FIG. 13. The SL spectrum is shown with center wavelength of 650nm. Over the bandwidth of the SL, the R pixels have the largestresponsivity, followed by G and then B.

Operationally, when SL illuminates mucosa, some light is scattered fromthe surface of the mucosa and some light penetrates the mucosa tissuesand experiences a combination of absorption and bulk scattering. Some ofthe bulk scattered light emerges from the mucosa some distance from thepoint of incidence. The visible SL spot is thus spatially broader thanthe light incident on the mucosa due to the diffusion of light in thetissues. This broadening or blooming could make it difficult todistinguish one spot from another.

FIG. 14(A) and FIG. 14(B) shows the sensor luma signal for R, G, and Bpixels as a function of position x on the sensor for the case FIG. 14(A)when an object is close to the endoscope and the irradiance on thesensor is high and the case FIG. 14(B) when an object is farther and theirradiance on the sensor is lower. The beams of light irradiating themucosa object are assumed to be uniform over a certain area whichappears in the image on the sensor as the regions of x over which thesignal is plateaued. Due to tissue bulk scattering, the irradiance onthe sensor does not fall off abruptly at the edge of these plateaus butextends more broadly over tails. The tails for two adjacent spots areshown to overlap. Diffuse background illumination from the SL source oranother source also may contribute to the signal and push it towardssaturation.

We note that the image sensor has a limited dynamic range and there is amaximum luma, luma-sat, corresponding to the maximum irradiance that canbe recorded, for a particular sensor gain. Luma-sat is determined by thesensor analog to digital converter (ADC). For the case of a 10-bit ADC,the maximum luma is 1023 digital counts. With continued reference toFIG. 14(A) and FIG. 14(B), we note that in FIG. 14(A) the R pixel lumaexceeds luma-sat and is saturated. Because the crossover point betweenthe two spots is saturated, the image of the two spots has merged into asingle spot so that the location of the two spots cannot be accuratelydetermined from the R pixel signal. However, the G and B lumas are notsaturated so that the spot locations can be determined from either orboth of these signals.

For the situation illustrated in FIG. 14(B) the signals are weaker andnone of the pixels is saturated. The R pixels have the bestsignal-to-noise ratio (SNR). Thus, the R signal is preferentially usedto determine the spot locations. Additionally, the image may have awhite-light signal (not shown) and the R channel is easier todistinguish from the white light image than the G or B as illustrated inFIG. 14(B). For the situation illustrated in FIG. 14(A) the saturated Rsignal can help to identify the presence of the SL spots in the presenceof a WL background, but the spot centroids are more accuratelydetermined from the G and B channels.

A sensor with pixels responsive to different color spectra, as opposedto a monochrome gray-scale sensor, increases the effective dynamic rangeof the SL detection if the response to the SL light is different butnon-zero for at least two of the color channels. The channels can becombined into a single channel of increased dynamic range or analyzedseparately. The example given is an RGB sensor, but other color channelscould be used such yellow, clear (white), magenta, cyan, violet, or IR.

FIG. 15 shows an illustrative capsule endoscope according to the presentdisclosure in a body lumen. The capsule has a tubular shaped middlesection with two hemispherical endcaps. At least a portion of thetubular wall is transparent. The endoscope includes a panoramic imagingsystem that images through the tubular wall with four cameras. Four lensobjectives face the tubular wall spaced approximately 90°. We note thatwhile this example includes four cameras and four objectives, thoseskilled in the art will appreciate that a greater or lesser number ofsuch elements may be employed so long as the desired FOV is achieved.Moreover, the same or similar apparatus, or a portion thereof, shown inFIG. 15 could be attached to an endoscope insertion tube at one end, asis shown for the apparatus in FIG. 9(A), yielding an insertion-typeendoscope with panoramic imaging including depth measurement.

In this illustrative system the FOV of the imaging system is 360° aboutthe capsule and from approximately 45° to 135° relative to thelongitudinal axis. Mirrors within the lens module fold the optical axesof the lenses. In a particular illustrative embodiment, images areformed on a common image sensor, which may have pixels in four separateregions. The capsule includes white light LEDs or other sources forilluminating the lumen wall.

As illustratively shown in FIG. 15, two rays from WL LED 1 and one rayfrom WL LED 2 are shown illuminating the lumen wall. One ray from WL LED1 reflects off mirror M2 before passing out of the tubular wall. A SLsource, such as a point source LED, emits light in a broad distribution.The light is spatially filtered and collimated into beams by the MLA.

Illustratively, the MLA includes microlenses arrayed in substantiallyconcentric rings such as that illustratively shown in FIG. 11. Themirror structure M1 comprises a plurality of annular reflective surfaceof various slopes. Advantageously, the surfaces could be conical orcould have shapes curved in two dimensions. M2 is—in this illustrativeembodiment—another annular mirror.

Shown further in FIG. 15 are chief rays of some of the beams thatreflect off three annular surfaces of M1. Some of the beams reflect offM1 and then reflect again off M2. Reflective surfaces of M2 are used toreflect both SL beams and white light illumination from WL LED1 anddirect both through the capsule housing to illuminate the lumen wall. M1and M2 could be injection molded plastic parts with aluminum coatings.Note further that also shown in FIG. 15 are chief rays of beams which donot hit any mirror and pass directly from the MLA through the capsulehousing.

As may be further observed in FIG. 15, a polyp is shown and illuminated,along with the lumen wall around it, by both white light, so that it maybe visualized, identified, and demarcated in an image captured by thecamera, and by SL, so that a depth map of the image may be generated,and the size of the polyp can be estimated. Beams at four differentangles relative to the longitudinal axis of the capsule cover the FOV ofthe camera on the capsule housing's outer surface. Both SL beam anglesgreater than and less than 180 degrees are produced so that thepanoramic field of view is covered. The beams travel both above andbelow a plane transverse to the capsule that includes the optical axesof the cameras. As noted previously, the number of cameras could be moreor fewer than four—depending upon their particular FOV and anyapplication requirements. Mirror M1 is shown with 3 conical annularsurfaces but that number too, could be more or fewer.

As noted throughout this disclosure, endoscope configurations in whichadditional optical elements follow an MLA in an optical path may afforddistinct advantages to those endoscopes. FIG. 16 is a schematic diagramshowing an illustrative configuration wherein such optical elementfollows the MLA. One particular advantage of such configurationsincludes the ability to increase the range of angles over whichstructured light is projected from θ1 to θ2, which may exceed 180degrees, as illustratively shown. The light source intensity may dropoff significantly beyond θ1 and the throughput of the MLA decreases withincreasing angle due to the foreshortening of the clear apertures andincreased Fresnel loss. Moreover, the MLA cost may be larger than theoptical element, so the arrangement of FIG. 16 minimizes the size of theMLA for a given lens focal length and desired FOV θ2. For endoscopeswith panoramic imaging, an optical element yielding θ2>180 degreesenables one structured light projector to cover the entire camerapanoramic FOV.

FIG. 17 shows an illustrative computer system 1700 suitable forimplementing methods and systems according to an aspect of the presentdisclosure. The computer system may comprise, for example a computerrunning any of a number of operating systems or embedded control orapplication specific control programs. The above-described methods ofthe present disclosure may be implemented on the computer system 1700 asstored program control instructions. As will be readily appreciated bythose skilled in the art, the specific computer system and componentsincluded therein may vary depending upon what specific aspect of thepresent disclosure is implemented thereon/therein.

Computer system 1700 includes processor 1710, memory 1720, storagedevice 1730, and input/output structure 1740. One or more input/outputdevices may include a display 1745. One or more busses 1750 typicallyinterconnect the components, 1710, 1720, 1730, and 1740. Processor 1710may be a single or multi core.

Processor 1710 executes instructions in which embodiments of the presentdisclosure may comprise steps described in one or more of the Figures.Such instructions may be stored in memory 1720 or storage device 1730.Data and/or information may be received and output using one or moreinput/output devices.

Memory 1720 may store data and may be a computer-readable medium, suchas volatile or non-volatile memory. Storage device 1730 may providestorage for system 1700 including for example, the previously describedmethods. In various aspects, storage device 1730 may be a flash memorydevice, a disk drive, an optical disk device, or a tape device employingmagnetic, optical, or other recording technologies.

Input/output structures 1740 may provide input/output operations forsystem 1700. Input/output devices utilizing these structures mayinclude, for example, keyboards, displays 1745, pointing devices, andmicrophones—among others. As shown and may be readily appreciated bythose skilled in the art, computer system 1700 for use with the presentdisclosure may be implemented in a desktop computer package 1760, alaptop computer 1770, a hand-held computer, for example a tabletcomputer, personal digital assistant or Smartphone 1780, or one or moreserver computers which may advantageously comprise a “cloud” computer1790.

At this point, while we have presented this disclosure using somespecific examples, those skilled in the art will recognize that ourteachings are not so limited. More specifically, our methods can befurther extended in that the structural events can embed more temporalinformation and consider more sophisticated structures includingconsidering more finegrained temporal information, e.g., the transitiontime distribution, to enrich mined structural events. Also, we havefocussed on transition relations among log patterns. There are otheruseful relations among logs, such as running in parallel that may beemployed. Those relations can be further modeled in the workflow graphusing undirected edges. We also believe that the methods according tothe present disclosure can achieve more utility in an interactivesetting, where system admins can interactively explore the systembehaviors with different focusses (parameter settings) on coverage,quality or connectivity.

Accordingly, this disclosure should be only limited by the scope of theclaims attached hereto.

The invention claimed is:
 1. A method for imaging a body lumencomprising: introducing an imaging apparatus into the body lumen;emitting, during a common integration period of the imaging apparatus,from the imaging apparatus, structured light and non-structured lightinto the body lumen; detecting, by the imaging apparatus, bothstructured light and non-structured light reflected from anatomicalfeatures in the body lumen; generating, by the imaging apparatus, imagesfrom the detected light.
 2. The method of claim 1 wherein each generatedimage is detected in an integration period, and the structured andnon-structured light from which the image is generated is detectedduring that integration period.
 3. The method of claim 2 wherein eachintegration period generates a single image frame of a video.
 4. Themethod of claim 1 wherein the non-structured light is white light. 5.The method of claim 4 wherein the structured light exhibits an opticalspectrum that is outside the white light spectrum.
 6. The method ofclaim 1 wherein the structured light exhibits an optical spectrum thatis different from the non-structured light.
 7. The method of claim 1wherein the imaging apparatus includes pixels that are transmissive tothe structured light and less-transmissive of the non-structured light.8. The method of claim 7 wherein the transmissivity of the imagingapparatus pixels are light wavelength-dependent.
 9. The method of claim1 wherein the imaging apparatus includes pixels that are configured toblock transmission of the non-structured light.
 10. The method of claim1 further comprising: before introducing the capsule endoscope into thebody lumen: projecting a structured light pattern emanating from thecapsule onto a scene; generating calibration data from that projectedpattern; and storing the calibration data into a memory.
 11. The methodof claim 10 further comprising capturing at least one image to generatesaid calibration data.
 12. The method of claim 11 further comprisingstoring the generated calibration date in a memory, wherein the memoryis located within the capsule endoscope.
 13. The method of claim 1further comprising removing detected structured light from the generatedimages.
 14. The method of claim 13 wherein the removing is performedprior to a displaying of the images.
 15. The method of claim 13 whereinimages from which reflected structured light has been removed aredisplayed as gray-scale image(s).
 16. The method of claim 1 furthercomprising: identifying an object of interest in a non-structured lightframe and structured light frame; and selecting a proximal structuredlight frame such that a size of the object may be estimated/determined.